One type of processing for generating images with higher resolution based on input images, is class classification adaptation processing. An example of class classification adaptation processing is processing wherein coefficients used in processing for generating images with higher resolution are generated beforehand, in the spatial direction, and images are generated with higher resolution in the spatial direction based on the generated coefficients.
FIG. 1 is a block diagram illustrating the configuration of a conventional image processing device for generating coefficients used in class classification adaptation processing for generating HD (High Definition) images from SD (Standard Definition) images.
Frame memory 11 stores input images, which are HD images, in increments of frames. The frame memory 11 supplies the stored HD images to a weighted averaging unit 12 and a corresponding pixel obtaining unit 16.
The weighted averaging unit 12 performs ¼ weighted averaging on the HD images stored in the frame memory 11, generates SD images, and supplies the generated SD images to the frame memory 13.
The frame memory 13 stores the SD images supplied from the weighted averaging unit 12 in increments of frames, and supplies the stored SD images to a class classification unit 14 and prediction tap obtaining unit 15.
The class classification unit 14 is configured of a class tap obtaining unit 21 and a waveform classification unit 22, and performs class classification of pixels of interest which are the pixel of interest in the SD images stored in the frame memory 13. The class tap obtaining unit 21 obtains a predetermined number of class taps which are pixels of the SD image corresponding to the pixel of interest from the frame memory 13, and supplies the obtained class taps to the waveform classification unit 22.
FIG. 2 is a diagram explaining the class taps obtained by the class tap obtaining unit 21. As shown in FIG. 2, the class tap obtaining unit 21 obtains eleven class taps at predetermined positions.
The waveform classification unit 22 classifies the pixel of interest into one class out of multiple classes, based on the class taps, and supplies a class No. corresponding to the classified class, to the prediction tap obtaining unit 15. The waveform classification unit 22 classifies the pixel of interest into one class out of 2048 classes, based on the eleven class taps.
The prediction tap obtaining unit 15 obtains a predetermined number of prediction taps which are pixels of the SD image, corresponding to the classified class from the frame memory 13, based on the class No., and supplies the obtained prediction taps and class Nos. to a corresponding pixel obtaining unit 16.
FIG. 3 is a diagram explaining prediction taps which the prediction tap obtaining unit 15 obtains. As shown in FIG. 3, the prediction tap obtaining unit 15 obtains nine prediction taps at predetermined locations.
The corresponding pixel obtaining unit 16 obtains, from the frame memory 11, pixels of the HD image corresponding to the pixel values to be predicted, based on the prediction taps and the class Nos., and supplies the prediction taps, class Nos., and the pixels of the HD image corresponding to the obtained pixel values to be predicted, to a normal equation generating unit 17.
The normal equation generating unit 17 generates normal equations corresponding to relationships between prediction taps and pixel values to be predicted, corresponding to the classes, based on the prediction taps, class Nos., and the obtained pixel values to be predicted, and supplies the generated normal equations corresponding to the classes, to a coefficient calculating unit 18.
The coefficient calculating unit 18 solves the normal equation supplied from the normal equation generating unit 17, calculates coefficient sets corresponding to each class, and supplies the calculated coefficient sets to coefficient set memory 19, along with the class Nos.
The coefficient set memory 19 stores the calculated coefficient sets corresponding to the classes, based on the class Nos.
FIG. 4 is a diagram explaining an overview of class classification adaptation processing. In class classification adaptation processing, a tutor image which is an HD image is used to generate a corresponding SD image, by ¼ weighted average processing. The generated SD image is called a student image.
Next, a coefficient set for generating an HD image from the SD image is generated, based on the tutor image which is the HD image and the student image which is the corresponding SD image. The coefficient set is configured of coefficients for generating an HD image from an SD image, by linear prediction and the like.
A quadruple-density image is generated from the coefficients set thus generated and the SD image, by linear prediction and the like. The processing for generating an image or the like with higher density, from a coefficient set and an input image, is also called mapping.
SNR comparison, or visual qualitative evaluation is performed, based on the generated quadruple-density image and a corresponding HD image.
A coefficient set generated from a particular tutor image and corresponding student image is called a self coefficient set of the particular tutor image and corresponding student image. Mapping using the self coefficient set is called self mapping. A coefficient set generated from multiple other tutor images and corresponding student images is called a cross coefficient set.
On the other hand, with images obtained by a video camera taking a foreground subject which moves across a predetermined stationary background, movement blurring occurs in the event that the speed of movement of the object is relatively fast, and mixing of the foreground and background occurs.
With conventional class classification adaptation processing, one set of coefficients is generated for all of the foreground, background, and portions where mixing between the foreground and background occurs, by learning processing such as described above, and mapping processing is executed based on the coefficient set.
The conventional learning processing for generating coefficients used in the processing for generating HD images from SD images will be described, with reference to the flowchart shown in FIG. 6. In Step S11, an image processing device judges whether or not there are unprocessed pixels in the student image, and in the event that judgment is made that there are unprocessed pixels in the student image, the flow proceeds to Step S12, and pixels of interest are obtained from the student image, in order of raster scan.
In Step S13, the class tap obtaining unit 21 of the class classification unit 14 obtains a class tap corresponding to the pixel of interest, from the student image stored in the frame memory 13. In Step S14, the waveform classification unit 22 of the class classification unit 14 performs class classification of the pixel of interest, based on the class tap. In Step S15, the prediction tap obtaining unit 15 obtains a prediction tap corresponding to the pixel of interest from the student image stored in the frame memory 13, based on the class into which classification has been made.
In Step S16, the corresponding pixel obtaining unit 16 obtains a pixel corresponding to a pixel value to be predicted, from tutor data stored in the frame memory 11, based on the class into which classification has been made.
In Step S17, the normal equation generating unit 17 adds a pixel value of a pixel corresponding to the prediction tap and pixel value to be predicted to the matrix for each class, based on the class into which classification has been made, the flow returns to Step S11, and the image processing device repeats judgment regarding whether or not there are unprocessed pixels. The matrixes for each class to which the pixel value of a pixel corresponding to the prediction tap and pixel value to be predicted are added, correspond to the normal equation for calculating coefficients for each class.
In the event that judgment is made in Step S11 that there are no unprocessed pixels in the student image, the flow proceeds to Step S18, wherein the normal equation generating unit 17 supplies the matrix for each class wherein a pixel value of a pixel corresponding to the prediction tap and pixel value to be predicted has been set, to the coefficient calculating unit 18. The coefficient calculating unit 18 solves the matrix for each class wherein a pixel value of a pixel corresponding to the prediction tap and pixel value to be predicted has been set, and calculates a coefficient set for each class.
In Step S19, the coefficient calculating unit 18 outputs the coefficient for each class that has been calculated, to the coefficient set memory 19. The coefficient set memory 19 stores a coefficient set for each class, and the processing ends.
FIG. 7 is a block diagram illustrating the configuration of a conventional image processing device for generating HD images from SD images, by class classification adaptation processing.
Frame memory 31 stores input images which are SD images, in increments of frames. The frame memory 31 supplies the stored SD images to a mapping unit 32.
The SD images input to the mapping unit 32 are supplied to a class classification unit 41 and a prediction tap obtaining unit 42.
The class classification unit 41 is configured of a class tap obtaining unit 51 and a waveform classification unit 52, and performs class classification of pixels of interest which are the pixel of interest in the SD images stored in the frame memory 31. The class tap obtaining unit 51 obtains from the frame memory 31 a predetermined number of class taps corresponding to the pixel of interest, and supplies the obtained class taps to the waveform classification unit 52.
The waveform classification unit 52 classifies the pixel of interest into one class out of multiple classes, based on the class taps, and supplies a class No. corresponding to the classified class, to the prediction tap obtaining unit 42.
The prediction tap obtaining unit 42 obtains from the input image stored in the frame memory 31 a predetermined number of prediction taps corresponding to the classified class, based on the class No., and supplies the obtained prediction taps and class Nos. to a prediction computation unit 43.
The prediction computation unit 43 obtains coefficient sets corresponding to classes from the coefficient sets stored in coefficient set memory 33, based on the class No. The prediction computation unit 43 predicts pixel values of predicted images by linear prediction, based on coefficient sets corresponding to classes, and prediction taps. The prediction computation unit 43 supplies the predicted pixel values to frame memory 34.
The frame memory 34 stores-predicted pixel values supplied from the prediction computation unit 43, and outputs an HD image wherein the predicted pixel values have been set.
FIG. 8 is a diagram illustrating the pixel values of the input image, and the pixel values of the output image generated by class classification adaptation processing. In FIG. 8, the white squares indicate input signals, and the solid circles indicate output signals. As shown in FIG. 8, the image generated by the class classification adaptation processing contains waveforms lost in the bandwidth restriction of the SD image. In this sense, it can be said that processing for generating an image with higher resolution by the class classification adaptation processing creates resolution.
The conventional processing for creating images, for generating HD images from SD image with an image processing device which executes processing for creating resolution by class classification adaptation processing, will be described with reference to the flowchart in FIG. 9.
In Step S31, the image processing device judges whether or not there are unprocessed pixels in the input image, and in the event that judgment is made that there are unprocessed pixels in the input image, the flow proceeds to Step S32, where the mapping unit 32 obtains a coefficient set stored in the coefficient set memory 33. In Step S33, the image processing device obtains pixels of interest from the input image in raster scan order.
In Step S34, the class tap obtaining unit 51 of the class classification unit 41 obtains a class tap corresponding to the pixel of interest, from the input image stored in the frame memory 31. In Step S35, the waveform classification unit 52 of the class classification unit 41 performs class classification of the pixel of interest into one class, based on the class tap.
In Step S36, the prediction tap obtaining unit 42 obtains a prediction tap corresponding to the pixel of interest from the input image stored in the frame memory 31, based on the class into which classification has been made.
In Step S37, the prediction computation unit 43 obtains a pixel value of a predicted image by linear prediction, based on the coefficient set corresponding to the class into which classification has been made, and the prediction tap.
In Step S38, the prediction computation unit 43 outputs the predicted pixel value to the frame memory 34. The frame memory 34 stores the pixel value supplied from the prediction computation unit 43. The procedures return to Step S31, and repeats judgement regarding whether or not there are unprocessed pixels.
In the event that judgment is made in Step S31 that there are no unprocessed pixels in an input image, the flow proceeds to Step S39, where the frame memory 34 outputs the stored predicted image wherein predicted values are set, and the processing ends.
Also, edge enhancing processing is used for converting the input image into an image with the sense-of-resolution enhanced even further. As with the class classification adaptation processing described above, the same processing is executed for the entire screen with the edge enhancement processing as well.
However, in the event that objects move in front of still backgrounds, movement blurring occurs not only due to mixture of the moving object images itself, but also due to mixture of the moving object images and the background images. Conventionally, processing images corresponding to the mixing of the background image and the image of the moving object had not been given thought.